Using the MPEG Query Format for Cross-Modal Identification

During the last years a vast number of multimedia databases have been established for search and retrieval of multimedia data, due to large audiovisual storage capacities and efficient compression methods. The metadata format of such databases can be based on the MPEG-7 standard. For the successful and efficient search, a query language for the interaction between client and database is crucial. One of the latest developments of the MPEG committee is the MPEG Query Format (MPQF) which is destined for the interaction between clients and MPEG-7 databases. The client may be interested to express more complex queries, which requires an interaction with several databases. Therefore an additional service provider can be used in the according environment, which accepts and understands a query formulation from a client and forwards parts of the query to one or more databases. The service provider furthermore retrieves the responses from these databases, postprocesses these results and replies a combined result list to the client. During the last years, the cross-modal search of visual and audio information has become more and more important. Using both domains, video and audio, turned out to be much more robust for the identification of video streams, than the visual part of the video stream alone. This paper describes a method for the audiovisual identification on remote databases using the MPQF. Additionally a service provider is deployed, which splits and aggregates the query and send them to two remote MPEG-7 databases (visual and audio) for identification. Among others a novel technique for the feature extraction on the service provider side is described, which is based on the MPQF. The interface between user and database is described in detail, examples are given and extensive results for the cross-modal search are presented.